In an industrial environment digital cameras are often used to monitor production processes, machines or objects, for example. Such industrial cameras are used for simple monitoring tasks and for metrological tasks such as for example quality control through image processing. They are characterized by their ease of integration into arrangements and high image quality.
FIG. 1 shows an exemplary application of a digital industrial camera. Digital industrial cameras 10 are intended for mounting in an industrial apparatus 23, as part of which they detect images of objects 14, 15 or e.g. persons. These images are transferred as image data 16, so that they can be evaluated and/or archived. The evaluation is often done automatically through an image processing device 18. In response to the results of evaluation automated actions are often performed, which are in context of the apparatus 23. In this way bright objects 14 and dark objects 15, which are distinguishable by their color, are automatically sorted based on optical criteria, for example. Such sorting is common in various industries. As an example, food, such as e.g. grains, is sorted based on the criterion of spotlessness, ore and rock based on its color or its luminance, postal items based on the address posted on them, brake discs or sealing rings based on the compliance with a correct shape and the compliance with certain desired dimensions, or empty returnable bottles based on a correct shape and the barcode attached thereon. Such sorting is often used for quality control or assurance.
Such industrial apparatuses 23 often comprise drive components which feed the objects 14, 15, e.g., a conveyor belt 11 with a drive motor 12 and a transport speed 13. The camera 10 captures the image at a suitable position by means of an optics 22, e.g., a lens. Thereby, the camera 10 can be configured as an area camera, a line camera or a multi-line camera, as described for example in EP 1 919 197 A2. The camera transmits image signals 16 of the captured images to an image processing device 18 (e.g. a computer (C)). Furthermore, an information 17 about a transport speed 13 of the conveyor belt 11 may optionally be supplied to the image processing device 18, which may for example be determined by a rotary transducer. Thereby, the image processing device can determine a suitable point in time, at which an object passes an actuator 19, e.g., an element with a plurality of controllable air valves, and can supply a control signal 24 thereto, based on which an action is performed, which is suitable to supply the corresponding object to a sorting. As an example, it can be specified by control of the air valves, whether the object falls into a first container 20 when the air stream is switched off or is deflected into a second container 21 when the air stream is switched on.
In such an application it is advantageous if the camera 10 can transfer the image as fast as possible, so that the image processing device 18 can produce the control signal 24 for the actuator 19 as soon as possible and thus the desired control action can be initiated as early as possible. Thereby, the apparatus 23 can be operated faster. In certain application cases, in particular if the objects 14 and 15 are moving on the conveyor belt 11, e.g. rolling, higher accuracy or precision can be achieved or other parameters can be optimized.
In other application cases, recording and/or documentation of a situation may be of interest. For example, images of vehicles may be recorded, which move irregularly in road traffic, e.g. when exceeding a speed limit or running a red light—as described for example in the DE 10 2010 003 039 A1. As an example, images of vehicles are also taken when ferry boats are loaded and unloaded, to determine whether a damage has occurred on the ferry boat in case of damage claims. Such applications require that the color of the objects and certain situations, e.g. red traffic lights, are reproduced correctly. Furthermore, it is important that the image is of good quality. This is among others the case, if the objects can be well recognized on the image and if for example writing, in particular such one with black and white contrasts, can be reproduced sharp and easily legible.
FIG. 2 shows a schematic representation of a structure of a digital camera 10 with a lens 22. An image scene 30 is imaged via the lens 22 on an image sensor 31 which comprises a regular arrangement of light sensitive elements (image points, pixels). The image sensor 31 transfers electronic data to a computing unit 32 mostly provided in the camera 10, which includes e.g. a processor, a digital signal processor (DSP) or a so-called Field Programmable Gate Array (FPGA). It may thereby be required to convert analog image data into digital image data, e.g., by means of an analog-to-digital converter (AD converter). In the computing unit the image data is converted into a form which is usable by a user and then output as an electronic signal 34 via an interface 33.
Camera speed and image quality, in particular, are of decisive importance for the performance of a digital (industrial) camera. The latter is determined according to the standard EMVA 1288 (www.emva.org) in the actual version 3.0. Furthermore, subjective human recognition of the image is applied by the user as important criterion for judging image quality.
Digital industrial cameras are used in various industries, e.g., food industry, logistics, transport, automobile industry, robotics and others. Within these industries a lot of different applications with different requirements are provided, e.g., as to the size of the image sensor, the optical resolution which is given by the number of pixels, as to the repetition speed measured in images per seconds (fps), and as to the data transmission format, e.g., according to the standards CameraLink, IEEE 1394, GigE Vision or USB. There are digital industrial cameras which achieve high data rates, such as for example up to 800 MByte per second with CameraLink or up to 100 MByte per second with GigE Vision. To guarantee efficient production at such a variety of requirements, such cameras are often built in a modular construction, e.g., with a sensor module comprising an image sensor, a computation module and an interface module with an interface. As the computation module shall adapt a lot of different sensor modules and interface modules, it is helpful if the software is configurable, so that the multiplicity of types of computation modules and the development, production and logistical effort associated therewith can be reduced. Furthermore, it is important to achieve a high data throughput for obtaining the required high data throughput. To this end, the computation module is often realized by an FPGA as central component.
The number of operations contained in a mathematical image processing method should be as little as possible. In case of an FPGA this leads to a low use of logical cells, which is advantageous to be able to use a cheap FPGA or make available more free logic cells of the FPGA for other functions. In case of a DSP or another processor, a low number of operations means that they can be applied on a higher number of pixels within a given time period, whereby an image output at higher data rate can be achieved. Also, a lower number of image processing operations allows use of a low performance and thus cheaper processor and thereby a reduction of the circuit complexity.
Monochrome cameras as well as multicolor cameras are used as digital industrial cameras. The most common method of capturing color images is the use of a single image sensor on which a mosaic filter is mounted. A mosaic filter consists of a two-dimensional arrangement of color filters which impart different sensitivities for different colors to the different pixels. The arrangement is often selected so that a filter basis is two-dimensionally periodically repeated.
FIGS. 3a-d show different color filter arrangements for mosaic filters with two-dimensionally repeating filter basis 40-43 of the size of 2×2 pixels.
An arrangement which is particularly often used for mosaic filters is the so-called Bayer pattern, which is known from the U.S. Pat. No. 3,971,065 and shown in FIG. 3a. It has a filter basis 40 of size 2×2 which is periodically repeated. The filters comprise colors red (R), green (G) and blue (B). The color green occurs twice as likely as the color blue.
For certain applications it might be required that the color infrared (Ir) is also captured in addition to the visible colors, e.g., R, G and B. The color infrared is not visible to humans, but can be captured by digital cameras. This is of great benefit e.g. in the food industry where, due to the four-dimensional red, green, blue and infrared (RGBIr) color information, bad old fruit can be much better distinguished from fresh fruit than would be the case on the basis of a red, green and blue (RGB) image. In the timber industry blemishes can be distinguished by RGBIr images from resinous parts of same RGB color, of which only the first ones affect the dead weight of a board or beam.
FIG. 3b shows a mosaic filter comprising the colors R, G, B and Ir, which has a filter basis (41) of size 2×2.
Furthermore, FIG. 3c shows a mosaic filter comprising the colors cyan (C), magenta (M), yellow (Y) and infrared (Ir), which also has a filter basis (42) of size 2×2.
Further mosaic filters with four different colors and a filter basis of size 2×2 are possible as well. As an example, FIG. 3d shows a mosaic filter comprising the colors R, G, B and white (W). The color white is sometimes also designated as “clear”.
When a mosaic filter is used, many applications require that a color image comprising several color values per pixel is generated from the raw data of the image sensor which comprises only one respective color value per pixel. Depending on the kind of mosaic filter, this process is called demosaicing, debayring or color interpolation. To this end, very often images with three color values per pixels are calculated, e.g., RGB, which can be displayed as color image on screens or printed by printers on paper. However, to achieve technical objects, e.g., for quality control, generation of images with more than three colors may be desired as well, e.g., with the four colors R, G, B and Ir.
Optionally, other information than color information may be allocated to single pixels or image areas. As an example, a three-color RGB image may be produced with a four-color mosaic filter, wherein an additional depth information may be allocated to individual image areas.
High-quality color interpolation is mainly characterized by good color reproduction in the resulting image. In this respect, it is particularly important that the colors of the imaged original are reproduced as genuine as possible. Here, known problems mainly occur at high spatial frequencies, e.g., at luminance or color edges. In particular, color fringes and color aliasing give rise to problems here.
Furthermore, color errors with a period of any two pixels are known, which are called zipper artefacts or zippering effects and which are recognized disturbing by a viewer.
Moreover, high quality color interpolation is characterized by a good structural reproduction. The structural reproduction is good for certain applications if it is faithful, which means if low and especially spatial frequencies are reproduced faithfully, e.g., by automated spatial measuring with sub-pixel accuracy. In other applications, structural reproduction is experienced as particularly good, if high frequencies are reproduced in an emphasized manner, whereby the image appears particularly sharp, e.g., for locating low-contrast edges. Here, known problems are a possible blur as well as a blocking artefact, which often occurs at a size of 2×2, when inferior color interpolation methods are used. Noise is experienced as very negative in the structure as well. On the one hand there are also specific applications in which a high signal-to-noise ratio (SNR) is provided, so that no disturbing noise is experienced, while it is important at the same time that fine details are visible as well, e.g., when thin film transistor screens (TFT displays) are checked. On the other hand, there are also certain applications in which fine details of minor importance are disturbing, while it is very important that the noise of the image is low, e.g., to achieve economical transmission bandwidths of compressed images.
Furthermore, it can be concluded that an image of a good digital camera is perceived as of high quality especially if it has a good signal quality, in particular a low noise and therefore a high signal-to-noise ratio (SNR) and is free of artefacts which a human viewer would experience as disturbing or which would influence computer algorithms in their function.
Typically, raw images are read out progressively from image sensors and are also transmitted progressively. However, for cost reasons it is desirable to keep storage requirements, bandwidth for memory access and computation effort as small as possible. Furthermore, applications require to achieve high image repetition rates and high data rates associated therewith. Industrial applications often require output of captured images at a low time latency, so that fast machines can be controlled by image processing.
It is an object of the invention to provide a camera having an image sensor with a mosaic filter, wherein a high quality color image with low latency and high image sharpness can be generated by color interpolation while preventing noise and disturbing artefacts.